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One Strategy for Assessing the Trustworthiness of Qualitative Research: 
Operationalizing the External Audit 

"The audit process became something like an independent study — or a synthesis experience 
bringing all my coursework, reading, and my first major research project together. ..all the while I 
collected and sorted data, I would periodically ask myself, ‘how will someone else find this? ’All 
the codes and categories that would be important for a 'stranger ’ really helped me to keep my 
work together. The necessity of organization here probably changed me forever — never had I 
been so careful to record and organize anything. These ‘skill sessions ’proved to be the 
‘work-out sessions ’for the ‘marathon ’ of qualitative data analysis. ..it was a tremendous 
confidence building experience ”. (-Kristin Anderson) 

Introduction : 

The concept of the external audit as one mechanism to assess the trustworthiness of 
qualitative research is not new to this audience. At the American Educational Research 
Association (AERA) annual meeting in 1982, Lincoln and Guba discussed the external audit at 
length in their paper, Establishing Dependability and Confirmability in Naturalistic Inquiry 
through an Audit . The following year at AERA, Halpem (1983) presented a paper on the audit 
process, and demonstrated its application to a qualitative case study. Later, in their books, 
Lincoln and Guba (1985) and Schwandt and Halpem (1988) expanded their discussion of the 
audit. 

The external audit assesses the trustworthiness of a study. Ultimately, the audit attests to 
the dependability of the study from a methodological standpoint, and the confirmability of the 
study by reviewing the data, analysis and interpretations and assessing whether or not the 
findings accurately represent the data. In essence, the audit examines both the process and 
product of the inquiry to determine its trustworthiness (Lincoln & Guba, 1982, 1985). 
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Though the concept of using the qualitative audit to assess the trustworthiness of an 
inquiry is dated, it has not been operationalized as a verification strategy extensively. In 1982, 
Lincoln and Guba suggested that audit reports attesting to the dependability of the study might 
be attached to inquiries that are submitted to journals for publication. This idea has not taken off. 
It has also not been extensively used at universities where students are conducting qualitative 
dissertations and the concepts of validity (or verification) and rigor are of paramount concern. 

In 1982, Lincoln and Guba suggested that few authors had offered specific suggestions 
regarding how to carry out the audit task. Fifteen years later, with the exception of Lincoln and 
Guba (1982, 1985), Halpem (1983), and Schwandt and Halpem (1988), this is still true. 
Numerous authors identify the audit as one strategy to determine the dependability and 
confirmability of the qualitative study (Creswell, 1994; Denzin, 1994; Ely, Anzul, Freidman, 
Gamer & Steinmetz, 1991; Erlandson, Harris, Skipper & Allen, 1993; Glesne & Peshkin, 1992; 
Lincoln & Guba, 1985; Marshall & Rossman, 1989; Maykut & Morehouse, 1994; McMillan & 
Schumacher, 1997; Meloy, 1994; Merriam, 1988; Miles & Huberman, 1994; Schwandt & 
Halpem, 1988; Yin, 1989). Yet, few provide more than a skeletal definition of the audit. Most 
authors cite Lincoln and Guba’s (1985) hallmark text, Naturalistic Inquiry as the primary source 
on the external audit, and for a discussion of qualitative validity issues that extend beyond the 
conventional discussions of validity and reliability (i.e. Lincoln and Guba use the term 
“trustworthiness” and suggest that qualitative research must be assessed by criteria more, 
appropriate to the naturalistic paradigm, and advocate using terms more appropriate to the 
qualitative paradigm; including credibility rather than internal validity, transferability rather than 
external validity, dependability rather than reliability, and confirmability rather than objectivity. 
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In our paper on Validity (Verification) in Qualitative Research: Perspectives, Terms, Procedures 
and Methodologies (Creswell & Miller, 1995) we discuss the concept of validity as applied to 
qualitative research and conclude that “indeed, ‘validity’ may not be a ‘valid’ term to use” (p. 

14). We use the term verification because we believe it “shifts the focus from tests, threats, 
instruments and data bases to perspectives and understanding” (p. 20). Thus, verification has 
been used throughout this paper. 

Lincoln and Guba (1985), Halpem (1983), and Schwandt and Halpem (1988) have 
provided a clear framework for the use of the external audit. This paper extends their discussion 
by illustrating how their framework has been operationalized by one auditor. My thinking about 
the external audit has evolved over the last six years. I was introduced to the audit in graduate 
school in 1991. 1 explored the concept while writing my dissertation proposal. I read sample 
audit reports, selected an auditor for my study (the only individual I knew who conducted 
audits), and throughout the research process was cognizant of the need to maintain a clear audit 
trail. In the fall of 1992, 1 began teaching a graduate qualitative research course at the University 
of Nebraska-Lincoln. Since that time, I have continued to study the audit process, and from 
April, 1994 to November, 1996 have audited eleven qualitative dissertations. In this paper, I 
draw on my experience conducting audits to discuss practical issues related to the audit process. I 
summarize the audit procedure I use and discuss some of the issues that arise in the audit 
process. 

Data are integrated from interviews with three faculty members at UN-L who recommend 
that students’ consider an audit, and eight former students (referred to as students throughout 
this paper) whose dissertations I have audited. Interviews were conducted on the telephone or by 
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e-mail. I asked faculty to describe their perceptions of the audit, why they recommend that 
students have an audit conducted, and how they view the role of the auditor. I asked students to 
discuss their perceptions of the audit and probed several issues: why they decided to have an 
audit conducted, how they prepared for it, what they perceived the value of an audit to be, what 
potential problems or pitfalls might be associated with the audit, how (or whether) the audit was 
discussed as part of the dissertation oral defense, and what they might advise students who are 
preparing to have an audit conducted. The data are integrated into applicable sections of this 
paper. Pseudonyms have been used to protect anonymity. 

Appendix A includes two audit attestations (used with permission). In addition, I 
analyzed the eleven audits that I have completed, and in Appendix B include a list of types of 
materials submitted for the audit. From the printed reports I summarized the types of concerns 
that I have noted as an auditor, and provide examples of those in a section related to auditor 
concerns. 

Several audiences might benefit from this practical discussion of the external audit 
process; faculty who are interested in the concept of the audit as a verification strategy, 
individuals interested in conducting external audits, and researchers who might consider using the 
external audit to attest to the trustworthiness of their work. 

The following questions frame this discussion of the external audit process: 

• What is an external audit? 

• Why use the external audit for verification? 

• Who should conduct an external audit, and what is the auditor’s role? 
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• What materials are submitted for an audit? 

• What are the steps in the audit process? 

• What does the auditor look for? 

• What kinds of concerns might the auditor identify, (and what happens when problems 

are identified)? 

• What is the final “product” of the audit, and how is it used? 

What is an external audit ? 

The audit is one of several verification strategies that may be used to assess the 
trustworthiness of qualitative studies. Lincoln and Guba (1982, 1985) describe the audit as a 
technique used to “authenticate” qualitative accounts and compare it metaphorically to the fiscal 
audit. It is an extensive examination of the researcher’s record-keeping, or the “audit trail” left by 
the researcher. It is a systematic evaluation, typically conducted by an external or independent 
auditor. 

The purpose of the audit is twofold: to examine the process and the product of the 
inquiry. The former addresses the dependability of the study, and the latter addresses the 
confirmability of the study. Lincoln and Guba (1985) suggest that the auditor examines the 
accounts that were kept, to satisfy readers that they are not victims of “creative accounting”, and 
that the data are fairly represented. The audit assesses whether the final account is accurate or 
“trustworthy”, and supported by the data. Once the auditor completes these tasks, he or she 
submits an “attestation” or “reviewer’s appraisal” that documents, in writing, the auditor’s 
conclusions. 
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An early task for any auditor is to determine whether a study is auditable; i.e., whether 
the researcher maintained a clear audit trail. McMillan and Schumacher (1997) describe this 
concept of “audibility” as “maintaining a record of data management techniques and decision- 
rules that document the ‘chain of evidence’ or ‘decision-trail’” (p. 410). They suggest that this 
documentation is critical because the integrity of the findings is rooted in the data themselves. 

The record links the findings to the data sources, and provides a chain of evidence that is open to 
inspection and confirmation by outside reviewers. 

Erlandson et al. (1993) suggest that the key to developing an audit trail is “reporting no 
‘fact’ without noting its source and making no assertions without supporting data” (p. 150). 
Goetz and LeCompte (1984) described the importance of an audit trail as enabling other 
researchers to “use the original report as an operating manual by which to replicate the study” (p. 
216). Yin (1989) suggests that “the general way of approaching the reliability problem is to make 
as many steps as possible as operational as possible, and to conduct research as if someone were 
always looking over your shoulder.. .an auditor is.. .performing a reliability check and must be able 
to produce the same results if the same procedures are followed. A good guideline for doing case 
studies is therefore to conduct the research so that an auditor could repeat the procedures and 
arrive at the same results” (p. 45). (Much discussion about the issue of the replicability of 
qualitative studies has occurred over the years, and most would argue that qualitative studies are 
not replicable in the traditional sense. However, the study methodology could be replicated and a 
clear audit trail would allow another researcher to do so.) 
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Why use the external audit for verification ? 

The audit process is particularly beneficial to researchers who seek validation for their 
work. The product of the audit is beneficial to readers who need to make some judgment about 
the trustworthiness of the findings. This was clearly reflected in the interviews I conducted with 
faculty and students. Most of the students had learned about the audit process in the 
introductory and/or advanced qualitative course on campus; three of the eight students recalled 
reading about the audit and thinking it was an “important step”; it was “one more check”. One 
student summarized: “I thought of it as a ‘piece’ of a well done dissertation”. In addition, five of 
the eight students noted that their advisor or committee member(s) had recommended that they 
have an audit done. One student explained: “the audit was recommended by my advisor, who 
clearly understood how helpful the audit could be to the strength of the whole dissertation”. 

Miles and Huberman (1994) suggest that though audits are not widely used, the most 
noted benefit is the “encouragement of systematic record keeping and reflexivity” (p. 439). Five 
of the eight students discussed the issue of record-keeping. They indicated that knowing they 
were going to have an audit done helped them “stay organized from the beginning”, made them 
“more accountable”, and the process more “rigorous”. Gina Matthews, Associate Vice President 
for Academic Affairs at a Midwestern liberal arts college suggested that the audit “guided my 
research from the beginning. The audit focused my record keeping and organization from the 
beginning, through this elongated process”. Because Matthews wrote her grounded theory 
dissertation over time, taking off three weeks here and there, she explained that it was critical that 
she be able to come back to it with a minimum amount of effort. The audit was “a very helpful 
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guide when I think of it in retrospect”. She added that the audit served as an “external 
conscience” because “you might as well do it right rather than take short cuts along the way. (The 
audit) kept me on the straight and narrow. Of course, your internal conscience says it is the right 
thing to do, but the audit forces a little more care, or perfectionism that is essential to the rigor of 
the study. You need that kind of care with qualitative research because you can lose that little 
nuance that is that keen insight”. John Mitchell, a music educator, explained: “just knowing that 
someone else was going to look at any piece of paper I wrote on, or computer file I generated, or 
journal entry.. .made me more accountable. I didn’t have to rely on my memory — I always told 
myself to write it out so that somebody could look at it and figure it out”. Mitchell added: 
“knowing someone was going to look over my shoulder kept me honest with the process. ..it is 
like stating a behavioral objective as a classroom teacher— knowing the goal makes it more rigorous 
in going there. I knew the audit was coming up, and that I had to do everything I could to leave a 
clear audit trail”. Kristin Anderson, now an Assistant Professor at a major university in Texas, 
prepared for the audit from the beginning: “all the while I collected and sorted data, I would 
periodically ask myself, ‘how will someone else find this?’ All the codes and categories of data 
that would be important for a ‘stranger’ really helped me to keep my work together. The 
necessity of organization here probably changed me forever — never had I been so careful to 
record and organize anything. These ‘skill sessions’ proved to be the ‘work-out’ sessions’ for the 
‘marathon’ of qualitative data analysis” One benefit, Anderson explained, was that “my 
expected audit kept me honest and rigorous as a researcher”. Amy Langston, site coordinator for 
a major technology grant in one school district, similarly suggested that the audit “kept me more 
honest and careful. ..(it) caused me to keep my things organized so I could present (my materials) 
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with an audit trail that you could follow”. Admittedly not a fan of journaling, trainer and 
consultant Cheryl Jefferson believed that “journaling helped during the process — each day I sat 
down and recapped — it is kind of like the painful process of analysis”. 



10 



Other benefits were clearly articulated in the interviews. The audit provided support and 
validation — that the findings were “trustworthy”, that the researcher had done all he or she could 
to validate the study, had been as “thorough as possible” and “covered all the bases”, and that the 
final project had “good integrity”. Ann Simon, an assistant principal at a middle school, 
described the audit as, “another opportunity for validating what you are doing — to know that 
you are on the right track.” Several said the audit gave them more “confidence”, and that this was 
particularly important because for most, this was a new process. Jefferson recalled that her 
advisor “didn’t really know a lot about qualitative research and couldn’t help me a lot.. .it’s such a 
lonely process. ..you don’t know if you are on the right track”. She continued, “it is a wonderful 
process for a student, a good learning experience. I had written the dissertation alone and there are 
a lot who have quantitative committees who don’t have anyone to help them through the 
process.. .it was reassuring to me”. Mitchell suggested that the audit gave him “greater security 
and peace of mind”. He added: “it put to rest my fear that, I’ve never done one of these things 
before, am I doing it wrong?” Matthews said that the audit was particularly important “because 
it was a new process for me”. She added, “I personally wanted the support of verification”. 
Anderson explained: “the audit process became something like an independent study — or a 
synthesis experience bringing all of my course work, reading, and my first major research project 
together. My auditor was my first real audience... qualitative research is ‘messy’ and I had made 
a considerable effort to organize the pieces of my data so that you might see the pieces as well as 
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the whole. You not only understood my work but commended my attention to detail and 
organization. It was a tremendous confidence building experience”. Marsha Bridges, Assistant 
Dean of Student Services at a medical school, wanted to have a “solid study” and viewed the 
audit as one “logical way to do that”. She said, “the audit gave me more confidence in my 
findings... I felt that the author validated what I wrote so that it didn’t appear that... I was 
reporting anything that was not in the data”. 

There was another dimension to this issue of confidence. Participants believed that the 
audit gave them greater confidence going into the oral defense. Five of eight said that the audit 
was discussed in the defense. Cheryl Jefferson recalled, “the audit made my orals a breeze 
because the committee knew my materials had been thoroughly checked... and the audit was 
discussed in the oral defense as one way to bring validity to qualitative research”. Sarah George, 
now a superintendent in the Southwest, suggested that the audit “can add a vote of confidence 
and heighten the level of comfort for a researcher going into that final phase of dissertation work, 
the committee’s review”. In fact, she added that for her, the audit’s “greatest strength was tied to 
the oral defense” because she expected “intense scrutiny” about the validity of qualitative 
research and the “technical review added strength”. I had noted a few concerns in her audit and 
George explained, “I had included a written response to the questions you posed in the audit 
document, all committee members saw and considered that additional explanation when analyzing 
my work before the oral defense. I believe the audit accounting put to rest any concerns the 
committee members might otherwise have had without it before they even arrived at the oral 
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defense”. 
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Several valued the audit as a mechanism for feedback; a way of getting another 
perspective. Matthews described the audit as “another lens”, Langston said it was “one more 
way to get input”. Simon believes it is “helpful to have that ‘extra voice’”. She added, “I needed 
direction if I strayed... I needed you to tell me to try another direction, or rethink what I had 
done”. Anderson explained that with suggestions from the auditor, “I was able to improve my 
writing and begin to feel capable as a qualitative researcher”. 

Some of the students indicated that they wanted readers to have confirmation that the 
study was trustworthy; that readers need to be able to assess whether “a piece of research is 
verifiable”. Generally readers do not have access to the data, and do not have a mechanism to 
make a detailed assessment of the trustworthiness of the study, so the auditor acts on behalf of 
the readers, inspecting and verifying the accounts, and attesting to the trustworthiness of the 
study (Lincoln & Guba, 1985). The audit was particularly crucial for students whose advisors, 
faculty readers and/or committee members were primarily quantitative researchers. At least four 
students addressed this directly. John Mitchell explained, “In my area of music education, 
qualitative research is really new... I think (the audit) helped to add another layer of validity to 
my study. ..(it) added another layer of comfort or ease for those who were uncomfortable with 
the qualitative process — and there were some of those on my committee”. Mitchell recalled that 
his committee members “weren’t suspicious” about the audit, but “were very positively taken 
by it.. .they were ‘wowed’ by it — totally impressed that a researcher would go to such lengths. 
They had never heard of an audit, it was the first qualitative dissertation they had ever read”. He 
also recognized that had the audit not been positive, “it would have raised some eyebrows, it 
would have raised double red flags”. Mitchell liked having an opportunity to “teach faculty 
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members something new”. Cheryl Jefferson recalled, “the majority of my committee were 
quantitative (researchers), and had some definite biases. It was very helpful to have done an 
analytical procedure”. Kristen Anderson related that her advisor was not familiar with qualitative 
methodology or the audit process. In addition, she explained, “One committee member, a 
botanist, was completely new to qualitative research and I knew an audit would help me answer 
her concerns about the limitations of my research”. “Looking back”, Sarah George recalled, “I am 
sure (my advisor) encouraged the external audit... she understood the importance of triangulation 
of data, intensive and thorough review of all research work along the way, but more importantly 
she recognized that she had limited experience with qualitative research. Hopefully other advisors 
with limited qualitative research competencies would similarly recognize the audit’s value”. 

These data support Lincoln and Guba’s (1985) claim that subjecting all of a researcher’s work 
and materials to an extensive audit can enhance the credibility of the findings, particularly for 
audiences that may not have a strong qualitative background, and may want extra assurances that 
the product is not fabricated. They concede that researchers have to come to terms with the 
doubt and skepticism that accompanies qualitative research and that the audit is one strategy that 
can be used to “persuade” readers that the account is trustworthy. 

Faculty indicated that the audit process is beneficial to the researcher. It provides another 
perspective on a subjective process, and can enhance and lend credibility to the final product. 
Christine Price, a professor who recommends her students consider an audit, suggested that the 
audit “gives the student another set of objective eyes, that strengthens what gets written”. She 
added, “the audit.. .makes a better research, writing and thinking experience”. Price believes that 
the audit process is particularly beneficial for the beginning scholar. As an advisor, her concerns 
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are “not whether the student is doing a good job, or authentic work, but it is a new and lonely 
experience. They benefit from the audit process”. From her perspective, the audit is “an 
intellectual strengthening... (it is a) methodological way to verify the accuracy and credibility of 
the account”. Price described the work the auditor does as different from what the advisor does. 
Most advisors are not going to sit down and examine all of the documentation the auditor 
reviews. In that sense, the audit provides a more holistic perspective of the research process. 

Alan Davis, who teaches proposal writing and advanced qualitative methods, believes that 
the audit process is probably most valuable “as a check for students”. The systematic record 
keeping required to have an audit done adds to the rigor of the study. It also identifies 
problematic areas and where adjustments can be made to produce a better product. Davis also 
views the audit as a “tool to help educate faculty, to highlight what was found, and identify 
adjustments that have been made during the study”. 

The external audit is one of the most systematic, rigorous verification strategies, and 
places specific demands on the researcher to keep good records, and establish a clear audit trail. 

It challenges the researcher to document the research process, and be intentional and careful about 
record keeping. It keeps them honest and accountable. However, it does much more. It provides 
support, validation and feedback, particularly through a lonely process. It gives researchers’ 
confidence in their work and abilities, and confidence going into the oral defense. It helps faculty 
who are uncomfortable or unfamiliar with qualitative research address some of their concerns 
about the validity of the process and product. It provides readers with a lens from which to view 
the credibility of the account. 
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Who should conduct an external audit (and what is the auditor’s role) ? 

Lincoln and Guba (1985) outline three criteria for an auditor. The auditor should be: 1) 
someone with the methodological expertise to conduct an audit of a qualitative study; 2) someone 
who has sufficient experience and credibility to be considered trustworthy, whose accounts will 
be considered valid by others; and 3) a disinterested party, who has no connection to or 
affiliation with the study, and no stake in the outcomes. They also suggest that it can be helpful 
to use someone who has knowledge of the topic. 

The issue of methodological expertise was noted by faculty and students. Christine Price 
has advised many students through the process of qualitative dissertations and has seen the 
product of at least three auditors. She concluded that having methodological expertise is critical 
and that some auditors are “more attentive to the integrity of the research and the whole process, 
and having a product that really supports the data”. She added that auditors “come from different 
perspectives and look at different things” and that most people can read and respond to students’ 
work, but “it is important to know what you are looking for, and how to carefully follow the 
researcher’s field notes and thinking”. Otherwise, “you may miss the methodological piece”. 
Then, Price suggested, the “research function” of the audit can be lost. 

David Shelby was cautious as he formulated his thoughts: “when you look at quantitative 
data, you can look at the numbers, and once you decide what statistical analyses to run, anybody 
can do it. But dealing with qualitative data is different — the auditor is able to give 
(methodological) assistance where something is so subjective and not only help the student avoid 
a mistake, but help him or her do it better”. 
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Four of the students raised this issue also. Cheryl Jefferson described a potential problem 
with the audit process as “having an auditor that does not understand what you are doing”. Using 
an analogy, Gina Matthews said that choosing an auditor is “like finding a babysitter — you don’t 
leave your child with just anyone”. She added, “it is very important to have some sense of who 
the auditor is — you have to trust the ability of that person”. Kristen Anderson indicated that 
problems could arise “if the auditor questions the trustworthiness of the study”, but at the same 
time adds that, “the researcher could question the trustworthiness of the auditor”. 

Generally the auditor is not involved in the study, and remains a disinterested party 
(Lincoln & Guba, 1985). However, in reflecting on my experience, I realize that I have developed 
a unique relationship with students whose work I audit. For many, I provide some degree of 
methodological support. Since I am one of a handful of professors who teach qualitative 
methodology at UN-L, seven of the eleven students whose work I audited were in my 
introductory qualitative class. Faculty recommended the others to me. Over the years I have 
developed rapport with students and faculty that has enabled me to assume a teaching role 
throughout the audit process. I am available to meet with students or talk with them over the 
phone, especially when they want to make sure that they are “on track” and that what they are 
doing “makes sense”. Sometimes they need another pair of “eyes” and to be able to process 
what they are seeing and thinking. Meloy (1994) discussed the need for “methodological support 
for doing qualitative research” (p. 71). She described the challenge for qualitative researchers as 
doing research that “is synonymous with multiple, simultaneous actions. The researcher as 
human instrument is a methodologist, analyst, writer, thinker, interpreter, inquirer — an individual 
human being capable of and responsible for some kind of final, organized presentation of the 
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interaction of experience in context... Novice qualitative researchers”, she concluded, “want some 
help in understanding what is possible and acceptable” (p. 71). 

David Shelby, one of the three faculty members I interviewed, explained, “I view your 
role almost as part of the instructional team. When you audit my students’ work, I’ve come to 
expect that, and I think it is a plus.” Shelby compared the process to auditing school district 
books. He suggested, “as an auditor, you might say, ‘what you’ve got is supported by the data, 
but maybe there is another way to look at it’... The records may be all laid out and may make 
sense, and the auditor may come in and say, ‘why don’t you organize this a bit differently’? The 
auditor is able to help you reconceptualize it.” Shelby provided an example: “the good thing 
about the audit you did with Barbara was. ..looking at her data you identified a fifth major 
category. You looked at materials I hadn’t seen and found another big idea... sometimes 
researchers have tunnel vision and need another perspective. In this case, Barbara would have 
missed something had you not prompted her to reexamine the data”. In reality, all I had done was 
question why, in the materials she submitted, she had begun with eight main categories, ended up 
with four, and clearly had a number of index cards with data chunks that appeared to relate to one 
major idea — I was simply trying to follow her audit trail. 

This relationship I develop with students through the audit process allows me to be 
candid with them about my concerns, and because I have some measure of credibility, students 
and faculty are willing to listen to those concerns and take time to address them. It also enables 
me to see more of the inquiry process, which heightens my awareness of the study methods and 
audit trail. 
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What materials are submitted for the audit ? 

Lincoln and Guba (1985) outline six categories of audit trail materials. They include 1) 
raw data, 2) data reduction and analysis products, 3) data reconstruction and synthesis products, 
4) process notes, 5) materials related to intentions and dispositions, and 6) instrument 
development information. I analyzed the eleven audits I have conducted. Appendix B includes a 
list of types of materials students have submitted for the external audit. I assigned materials to 
Lincoln and Guba’s six categories and proposed two additional categories: 7) validity/verification 
documentation, and 8) materials specifically prepared for the auditor/audit process. 

At our first meeting, I discuss the types of materials that students need to submit for the 
audit. My advice is to 1) keep good, dated records, i.e., document and save everything ; 2) always 
consider that someone will need to be able to follow the audit trail by simply reviewing available 
documentation, and 3) organize, label and catalog materials clearly before they are submitted for 
external review. I encourage students to create a research log or journal that documents all 
research activities, personal reflections about the research process, hunches about emerging 
themes, interpretations, and research decisions and rationales. 

Three examples of materials that have been particularly helpful to me during the audit 
process include: 

1) a memo generated by one researcher, consisting of several pages of typed, single spaced notes 
reflecting on processes and research decisions that may not have been clearly evident in the 
documentation; 2) an extensive research log, created by another researcher to recorded everything 
he did throughout the research process, from meeting with his advisor/committee, to 
correspondence, phone calls, reading content or methodological literature, analyzing data and 
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organizing the chapters of the dissertation; and 3) the submission of an extensive audit trail of a 
student’s emerging grounded theory including a box labeled “The Discovery thru Writing 
Process”. This included 10” of dated drafts and revisions of dissertation sections and chapters, 
and evolving theoretical propositions, matrices and narrative discussions. These materials helped 
me follow the “chain of evidence” that McMillan and Schumacher (1997) refer to. 

What are the steps in the audit process ? 

The following is a summary of the steps that I use when conducting an audit: 

1) 1 negotiate the audit with the student. This includes explaining the purpose and procedures of 
the audit, outlining the types of materials that need to be submitted, sharing sample audit 
attestations, discussing the timeframe for the audit, providing an estimate of the time and cost 
generally spent conducting an audit, and determining whether the audit is something both parties 
want to pursue. This negotiation is best done at the beginning of the study so that the student is 
aware of expectations for record keeping. 

2) If the student decides to have an audit conducted, I request a copy of the study proposal when 
it is approved by the faculty committee. When I receive the proposal I set up a file on the 
student and place a notation on my calendar regarding the audit. 

3) The student delivers all materials for the audit, with any special instructions that might be 
helpful in guiding me through their materials. 

4) I read the study proposal, particularly concentrating on the research questions and proposed 
methodology. I record the purpose statement, research questions, data collection and analysis 
strategies — including sampling, and verification strategies, in a notebook. 
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5) I compare the proposed research questions and methods with those outlined in the study 
(dissertation). I note whether or not there have been changes however, do not typically view this 
as problematic. Since qualitative research is an emerging design, change might be expected as the 



study evolves. I primarily look for documentation of a solid rationale for making changes. 

6) I catalog all materials submitted by the student (every file, transcript, tape, field note, note, 
draft, photograph, document). I record all items in detail, with labels and dates whenever the 
student has included those. I record numbers of pages of materials in some cases (transcripts, 
journals, field notes). 

7) Once this catalog is complete, I compare it to the catalog provided by the student and identify 
any discrepancies. I record all discrepancies in a notebook to later address with the student. 

8) I listen to excerpts of interview audio tapes, and check them against pages of transcripts for 
accuracy of transcription. I record notations of inaccuracies (both number of and types) per page 
examined. 

9) I read excerpts of transcripts, field notes, and journals so that I have a good sense of the data. 

10) I check all verification materials and pay close attention to participant and peer feedback. 

11) 1 examine all analytic procedures and try to trace evolving themes/theories. I also examine 
how analytic codes, themes, categories, and/or theories were translated to the written product. 

12) I read the dissertation draft, concentrating on how data are presented and conclusions and 
interpretations are framed. I compare this to the materials reviewed and record strengths, 
concerns and questions. Having viewed all of the materials at this point, I record my conclusions 
regarding the audit trail and trustworthiness of the study. 
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13) I contact the student for clarification and to provide feedback regarding my conclusions. 

14) I formulate a preliminary draft of the audit attestation and submit it to the student for review. 
At this point, the student has the opportunity to address concerns noted and submit further 
documentation that those concerns have been addressed. 

15) The final audit attestation is revised, signed and submitted to the student. All materials are 
returned to the student and we bring closure to the audit process. 

The audit typically occurs near the end of the study, so that all records can be compiled 
and accounted for. I ask students to deliver materials to me at the same time that they submit 
their first full draft of the dissertation to their faculty advisor. Examining the final product is an 
important part of the audit process, so the audit comes as the student is nearing completion of 
the research process. I complete the audit while the advisor is reading the first draft, and make 
suggestions for revisions so they can be incorporated into the next draft that the readers will 
receive. 

The amount of time it takes to conduct an audit depends on how extensive the materials 
are, how organized they are, and how many concerns or questions the audit raises. If students 
choose to address concerns and I revise the audit attestation, this elongates the process. On 
average, I spend 13 to 18 hours auditing a qualitative study. Once I receive materials from 
students, I try to complete the audit within two weeks. 

What does the auditor look for ? 

When I conduct an audit I assess whether or not the research process is clearly 
documented, rationales for research decisions are clearly presented, subjective and intuitive 
processes have been recorded, data seem to accurately represent participants’ realities, analytic 
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structures are grounded in the data and linked to the final product, and conclusions and 
interpretations are supported by the data and warranted, based on the documentation provided. 
Though documentation is provided, to some extent some of these auditor decisions are subjective. 
As a result, I carefully scrutinize the researcher’s materials, make detailed notes and ultimately 
try to answer two questions: 1) has the student left a clear audit trail that other researchers could 
follow, and 2) based on all the materials submitted, is there evidence that the results of the study 
are trustworthy? 

Schwandt and Halpem (1988) pose a number of key questions that are helpful in guiding 
this process: 

• Are findings grounded in the data? 

• Are inferences logical? 

• Is the category structure appropriate? 

• Can inquiry decisions and methodological shifts be justified? 

• What is the degree of researcher bias? 

• What strategies were used for increasing credibility? 

Miles and Huberman (1994) expand on these and suggest additional questions: 

• Are the study’s general methods and procedures described explicitly and in detail? 

• Can we follow the actual sequence of how data were collected, processed, 
condensed/transformed, and displayed for specific conclusion drawing? 

• Are the conclusions explicitly linked with exhibits of condensed/displayed data? 

• Is there a record of the study’s methods and procedures, detailed enough to be followed as an 
“audit trail”? 
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• Has the researcher been explicit and as self-aware as possible about personal assumptions, 
values and biases, affective states — and how they come into play during the study? 

• Are the research questions clear, and are the features of the study design congruent with them? 

• Is the researcher’s role and status within the site explicitly described? (note: this is sometimes 
discussed at length in the methods chapter in a section titled, Role of the Researcher) 

• Are basic paradigms and analytic constructs clearly specified? 

• Were data collected across the full range of appropriate settings, times, respondents... as 
suggested by research questions? 

• Were any forms of peer or colleague review in place? ( note: I pay particular attention to all 
validity/verification strategies used). 

I use these questions as a cognitive checklist when I am performing an audit and rely on 
them to help me frame my conclusions regarding the trustworthiness of the findings and 
interpretations. 

What kinds of concerns might the auditor identify, and what happens when problems are 
identified ? 

As I examine the student’s materials I record concerns, formulate questions for the 
student to respond to, and construct the initial audit attestation. If there are substantive concerns, 
the student has the opportunity to address those before the final audit attestation is prepared. 
The types of concerns that I have noted typically fall into three categories: 

1) minor discrepancies or issues that need clarification or elaboration (such as unclear rationale for 
researcher decisions, discrepancies on document dates/labels, intuitive processes not clearly 



documented); 
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2) substantive issues that warrant that the student return to the data, records and/or manuscript 
and make adjustments (such as conclusions not supported by the data, themes or categories not 
identified, unbalanced perspectives represented, lack of analytic process and/or structure); and 

3) procedural or methodological concerns regarding the research design, that may have impacted 
the data collection process and interpretations (such as the failure to explore or probe 
contradictory or negative information, inconsistent use of interview protocols/questions, the use 
of closed-ended or leading questions, decisions to change data collection procedures without a 
sound rationale). 

Minor discrepancies or concerns can generally be addressed easily. More substantive 
issues may also be addressed if the researcher is willing to reexamine records, make adjustments, 
and rewrite conclusions. Procedural or methodological problems that were inherent in the research 
design are generally problematic and can not be addressed short of returning to the field. At this 
late stage, most students do not have the time to do this. Concerns that are not addressed are 
noted in the audit attestation. 

The following are examples of concerns noted in audit attestations that students were able 
to address: 

• research questions were not clearly articulated — they were phrased in statement form, 
interspersed throughout the introduction, and unfocused. This clearly impacted discussion of 
conclusions in the final chapter of the dissertation; 

• inaccuracies in transcription that were included in the written product; 

• data analysis procedures were skeletal or unclear; i.e., themes were not fleshed out, the 
relationship between codes, major and minor themes, and categories were not clear; 
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• the structure of the dissertation did not match the analytic structure and identified themes; 

• conclusions were clearly not grounded in the data; the student made intuitive leaps and/or got 
off on tangents. 

The following are examples of concerns noted in audit attestations that were not 
addressed: 

• the student found the use of a semi-structured interview protocol to be too restrictive so rather 
than restructure the protocol, abandoned its use altogether and asked general questions to get 
participants to talk. This worked generally, however, since some key questions were not asked 
consistently, therefore, data were not as complete in some areas and the audit trail was not as 
clear; 

• in soliciting participants’ perceptions in one particular study, the student did not ask consistent 
questions. In the dissertation she included a section discussing participants’ perceptions 
regarding advantages and disadvantages of a particular issue. However, only two of the four 
participants’ views were represented in the dissertation and they both cited the advantages. In 
reviewing all of the transcripts, I discovered that the other two participants were not asked this 
question, so the perspective represented seemed unbalanced. This issue also arose in other 
sections of the final study — only two of the four participants were represented in the written 
product; 

• the student did not keep careful records — he conducted three site observations but did not 
record field notes, he began but did not maintain a journal, member checks for verification were 
conducted with only two of 21 participants (no effort was made to contact the other 
participants, and no other verification strategies were used); 
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• though the student had collected over 2,000 pages of rich data from 45 interviews and 10 
observations, she summarized each major theme with a single quote (when I discussed this with 
her I learned that it was the advisor’s decision to remove the majority of the quotes that she had 
originally included as “evidence”. I understood the dilemma she faced. I knew from the 
documentation that she had provided me — multiple files and boxes worth — that she had 
established a clear audit trail and attested to this. However, I cited this concern as a potential 
weakness because I believed that it hindered the readers’ ability to make decisions about the 
applicability of the finding to their settings and to judge the trustworthiness of the study); 

• the student proposed to interview six participants (high school students) a minimum of three 
times each, totaling at least 18 interviews. With no evidence of a sound rationale, she interviewed 
each participant once and conducted three interviews in pairs (resulting in a total of 9 interviews). 
Though she supplemented the interviews with observations and student journals, she ended up 
with a total of 1 05 pages of raw data for her dissertation (including interview transcripts, 
observation notes and journals); 

• throughout her interviews, the student used dichotomous (yes-no) closed-ended questions 
without the use of probes to follow up participant responses in depth. For example, I noted in 
the audit that the student asked high school seniors “does this class seem easier than the class 
you took last year? when a better way to ask the question might have been, “How would you 
compare the amount of work in last year’s class to this year’s class?”. She asked, “Are your 
teachers using different methods of teaching than they did last year?” I suggested that it might 
have been better to ask students to describe a typical class last year, then a typical class this 
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year, and ask them to compare the two related to teaching methods. This style of questioning 
resulted in minimal data, which significantly impacted the written product; 

• failure to probe for contradictory information that might challenge positive findings — for 
example, the student asked about differences in classes between last year and this year: — the 
participant said, “in some classes it’s (a new class scheduling structure) a lot better and in some 
classes it’s a lot worse”. The researcher changed the subject and this contradiction was not 
explored anywhere in the interview process. 

The first question I posed earlier in this paper, ‘did the researcher leave a clear audit 
trail?’, is straight forward, and can be easily determined from the documentation provided by the 
researcher. The second question is more problematic. Though the researcher may have clearly 
documented every research decision and activity, the auditor may question the trustworthiness of 
the study because of decisions the researcher made that call the objectivity or rigor of the study 
into question. 

When significant concerns arise, I document them on the audit attestation. Copies of the 
final attestation are provided to advisors and readers (by students) and they make a decision 
whether or not to follow up on the concerns. 

Two students specifically addressed this issue of responding to auditor concerns. Marsha 
Bridges completed a solid grounded theory study. She admits though that as a result of the audit, 
she “had to double check some things and re-do some things” and “one could argue that (the 
audit) slows your project down — but the trade off is quality, and in my opinion, there is no trade 
off where quality is concerned”. Sarah George explains, “if the results of (the audit) are not 
good.. .it provides very valuable information about where research needs to be shored up before 
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taking it to a larger group. In other words, it provides a ‘heads up’ to any weaknesses. Of course, 
this can mean extending the preparation and writing and rewriting time, but better that than being 
subjected to a fire storm at the oral defense”. 

What is the final “product” of the audit, and how is it used ? 

At the completion of the audit process I formulate an audit attestation (Appendix A). 

The attestation is generally 5 to 7 pages, single spaced. It provides an introduction to the concept 
of an audit, outlines all materials reviewed for the audit, documents steps of the audit process, 
identifies concerns, summarizes the auditor’s conclusions, and makes statements regarding the 
audit trail and trustworthiness of the study. The attestation documents the student’s name, title 
of the study, date the audit was conducted, and the auditor’s signature. 

Since the eleven studies I have audited have been dissertations, the audit has been 
referenced in the methods section of the dissertation and included as an appendix. Faculty 
advisors generally pay attention to the audit, however it may or may not be addressed in the oral 
defense. Christine Price, a faculty member in Education, suggests “the audit is part of the 
appendices, and faculty read them. If consistencies or concerns have been raised, we expect that 
those issues have been addressed... the audit is important — it is not ignored”. Price adds that 
students contract for the audit at the front end of the study (during the proposal stage), and it is 
“expected at the back end”. 

Davis provides a little different perspective on the use of the audit in the oral defense. “It 
depends on the timing of the audit, when the readers receive it and whether or not they have had 
time to look at it”. He suggests that in some cases, “faculty have little understanding of why it is 
being done and what its value is”. Price finds himself, often as the primary faculty member with 
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qualitative expertise on a student’s committee, being the one to raise issues in the oral defense. 

He is the one who suggests, “let’s go over these points in the audit”. He adds that “students 
don’t generally talk about it”. 

Conclusion 

This paper provides a limited perspective on the external audit process (the perspective 
of one auditor, students and faculty on one campus). It demonstrates the operalization of the 
audit concept, however, there are several issues it does not address. This paper does not examine 
the criticism that the audit is a quantitative concept applied to a qualitative process in an attempt 
to justify qualitative research (when the qualitative community is trying to move past the 
qualitative/quantitative comparison). It also does not address the question, “how important is 
validity as an external standard?” when there are scholars, such as Wolcott (1994), who view 
validity as a distraction. Wolcott suggests that “validity neither guides nor informs” his work (p. 
356). Though he essentially rejects the notion of validity, Wolcott describes strategies he uses to 
go “to considerable pains not to get it all wrong” (1990, p. 127). Most ethnographers do not 
seem to use an external standard like the audit to address the trustworthiness of their research. 

Dilemmas the auditor faces are not addressed in this paper. For example, what happens 
when the auditor determines that the researcher has not left a clear audit trail, and raises serious 
concerns about the trustworthiness of the findings. This can be difficult. On one hand, the 
researcher is paying for auditor services, only to have the auditor find fault with the study. On 
the other hand, the auditor is hired to objectively make decisions about the rigor and quality of 
the study. 
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This paper also does not focus on students’ concerns about the audit process. However, 
in the interviews I asked students to share their perceptions about problems associated with the 
audit process. Though they indicated that “the benefits far outweigh the problems”, five general 
concerns were raised: 1) the time it takes to prepare materials for the audit, which may vary 
depending on organization throughout the process (Gina Matthews explained that preparing for 
the audit was “one more entire day in the process”. ..because it is “time consuming to organize 
your materials so that someone else can make sense of them”; Amy Langston recalled: “the good 
news was, the audit made me stay organized. The bad news was, the audit made me stay 
organized! And that takes extra time”); 2) time constraints in completing the audit process 
(getting materials to and from the auditor, addressing auditor concerns, getting the audit 
attestation to faculty advisors/readers in a timely fashion, while meeting university deadlines); 

3) the cost associated with having an audit conducted (an average of $150 to $250) 

(though Kristin Anderson cited this as a concern she had an interesting perspective on it — “in 
my case” she explained, “the audit proved to be the most valuable qualitative learning experience 
of all, and cost less than any of my three hour courses or my new personal qualitative library”; 
Marsha Bridges suggested that researchers “include the cost of an audit in their budget”); and 

4) auditor access — finding someone with methodological expertise whose has credibility and who 
the researcher can trust (two students in particular expressed concerns about finding auditors on 
their campuses — Kristin Anderson said, “I will recommend the audit process to my doctoral 
students, however, I don’t know who here would audit their research”); 5) the concern that the 
auditor might note significant problems in the audit (Gina Matthews expressed that “on one 
hand, the audit is another worry... not only do you have to answer to your committee, but you 
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also have to meet the standards of the auditor”; Marsha Bridges suggested that it is helpful to 
“consult with the auditor in the beginning and throughout the study” so that expectations are 
clear along the way). 

Finally, the audit is an external assessment that may not be necessary or appropriate for 
all studies (this paper does not address the issue of confidentiality of materials — faculty member 
Christine Price explained that she has a student who would benefit from the audit process, who 
needs an outside perspective, but because her study is so sensitive, can not release her materials 
to an auditor). 

This paper demonstrates how one auditor has operationalized the external audit 
procedure. Students and faculty have indicated that the audit can be a valuable tool. It 
necessitates organization and systematic documentation of research activities and decisions. The 
audit may be particularly useful in addressing quantitative faculty/reader concerns. The process 
may be especially valuable for new researchers who are finding their way on a lonely journey. 
The confirmation they receive from the audit can provide greater confidence in the qualitative 
process, in the results that they have produced, and in their abilities as researchers. However, the 
audit is only one of several verification strategies that researchers may use to verify that their 
findings are trustworthy. Not all qualitative studies are auditable and not all researchers recognize 
this external standard as a valuable part of the qualitative process. 
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Attestation 

by Dana L. Miller, Ph.D. 

Julie Thomas requested that I complete an educational audit of her qualitative 
dissertation titled Resistance to Reform: A Critical Ethnography of Elementary Science 
Teachers . The audit was conducted between March 19-27, 1995. The purpose of the 
audit was to ascertain the extent to which the results of the study are trustworthy. 

In their book Naturalistic Inquiry, Lincoln and Guba (1985) suggest that the audit 
"may be the single most important trustworthiness technique available to the naturalist" 
(p. 283). The educational audit is "based metaphorically on the fiscal audit" (p. 317). 
The role of the auditor is to carefully examine both the process and product of the 
inquiry. In order to accomplish an audit, it is imperative that the researcher maintain 
careful, detailed records throughout the inquiry. 

Lincoln and Guba (1985) delineate two tasks in the audit process: 

1) examination of the PROCESS of the inquiry to ensure that informants are represented 
fairly in recorded accounts, and 2) examination of the final PRODUCT to ensure 
accuracy; in particular that the findings are supported by the data. 

To meet the outlined purpose of this audit the following materials were reviewed : 

1) The dissertation proposal, dated September 1994. 

Particular attention was paid to sections addressing the purpose of the 
study, research questions, critical ethnographic design, selection of 
informants, proposed data collection and analysis methods, verification 
strategies, the researcher's role in the study, and the interview, 
observation and dialogical reflection protocols. 

2) Forms requesting permission to conduct the study, participant consent forms 

and the Institutional Review Board approval form. 

3) Coded transcriptions of interviews (organized chronologically) and 

corresponding audiocassette tapes including interviews with secondary 
informants dated 4/6/94, 5/5/94, and 5/20/94; and interviews with primary 
informants including interviews with Krista dated 3/15/94, 5/2/94, 5/25/94; 
interviews with Jenny dated 3/3/94, 5/5/94, 5/26/94; interviews with 
Susan dated 3/10/94, 5/25/94; and interviews with Tom dated 3/22/94, 

3/23/94, 5/26/94. Transcripts were coded in right margins, using at least 
two coding schemes, then color coded with tabs to correspond to three 
main categories: 1) environment, 2) support, 3) vision. 

4) Field notes (organized chronologically) with analytical notes/codes in the 

right margin and color coded tabs to indicate correspondence to three 
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Attestation for J. Thomas-Page 2 

major categories. Thirteen field entries were submitted, dated 2/24/94, 
3/24/94, 4/14/94, 4/28/94, 5/2/94, 5/5/94, 5/6/94, 5/9/94, 5/16/94, 5/17/94, 
5/1 8/94, 5/25/94, and 5/26/94. 



5) Teacher observations (handwritten and typed sets, organized 

chronologically) with codes and color coded tabs to correspond to three 
main categories. Twenty one observations were submitted including: 
nine observations of Tom, dated 3/15/94, 3/23/94, 3/24/94, 4/12/94, 

4/28/94, 5/12/94, 5/13/94, 5/20/94, and 5/26/94; eight observations of 
Jenny, dated 3/23/94, 3/24/94, 4/12/94, 4/28/94, 5/5/94, 5/12/94, 5/13/94, 
and 5/16/94; and four observations of Krista/Susan, dated 3/24/94, 

5/12/94, 5/16/94, and 5/20/94. 

6) Transcripts of dialogical reflections (organized chronologically, coded and 

color coded with tabs to correspond to three main categories) with 
corresponding audiocassette tapes. Four dialogical reflections were 
conducted, dated 5/5/94, 5/13/94, 5/17/94, and 5/24/94. 

7) Artifacts students volunteered to share (experiments related to how far various types 

of gum stretch-students recorded hypotheses, results, conclusions on poster 
paper). 

8) One notebook filled with photographs of classrooms and students, taken 

during the fieldwork experience. 

9) Statewide Systemic Initiative (SSI) publications (color coded blue to 

correspond to environment category). 

10) School publications (color coded blue to correspond to environment 

category). 

1 1) Lesson plans and student work pages (with analytical notes and color coded tabs to 

correspond to three main categories). 

12) Mimifest panel notes and Peter Mars ten White Paper. 

13) Local news articles (color coded blue to correspond to environment 

category and yellow for vision category). 

14) Commercially prepared tape about SSI. 
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15) Vail (Wolcott/Creswell) conference notes. 

16) Jim Thomas Correspondence dated from 9/22/93-1/14/94. 

17) Grade 3 journals (on topics including: symmetry, weather, water measure, 

recycling, simple machines, decimals, magnets, car ramp, other). 

1 8) Member checks of chapter drafts (Chapters IV, V, VI) and discussion 

regarding Chapter VII. 

1 9) A one page matrix describing the four primary informants' background 

(years of experience, degree, license(s), professional organizations). 

20) A chronology of data collection. 

21) Protocols for personal history interviews, field notes, secondary informant 

interviews, and dialogical reflection. 

22) A list of all data reviewed, with each entry categorized, dated and 

organized chronologically. 

23) An extensive outline of the category scheme for data analysis. Codes and 

their locations were organized/listed by three main categories 
(environment, support, vision). 

24) A file folder of handwritten pages, labeled "Evolution of Chapter Outlines". 

25) A list of materials included in the audit trail. 

26) The first draft of the dissertation, dated March 1995. 

The audit consisted of the following steps : 

1) I read the research proposal in its entirety. 

2) I read a folder Julie had prepared for the auditor, including a note 

introducing me to the materials, a list of all materials compiled for the 
audit trail, a chronology of data collection, protocols used, a list of all data by 
category, a list of data reviewed, the category/theme scheme for data 
analysis, and a collection of notes on the evolution of the dissertation 
chapters (IV-VII). 
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3) I cataloged and reviewed all materials submitted for the audit. 

4) I checked the chronology of data collection (Appendix A) against the catalog 

of materials submitted for the audit, then checked the list of data reviewed 
(Appendix G) against the catalog of materials submitted for the audit. 

5) I listened to excerpts of taped interviews and reviewed the corresponding 

transcripts to assess the accuracy of transcription. 

6) I read the dissertation chapters that related to the findings and conclusions 

of the study (Chapters IV- VII). 

7) I compared the original dissertation proposal (focusing on the purpose of the 

study, research questions, proposed methods) to the dissertation. 

8) I reviewed chapter three again, focusing on research methods, analysis, and 

verification procedures. 

9) I met with Julie to discuss the audit procedures and conclusions. 

1 0) I wrote and submitted the attestation. 

Following completion of the preceding steps, this auditor submits the following 
conclusions : 

1) It is this auditor's opinion that the focus of the study remained consistent with the 
proposed focus. Julie did, however, slightly reframe her research questions as it became 
apparent to her that she was not studying teacher resistance (as she had 
expected to see), rather resistance to change in the school culture. This revision is to be 
expected since qualitative research is an emerging process and initially research questions 
are tentatively posed. In addition, data collection and verification strategies followed 
those proposed in the methods section of the proposal. Data analysis procedures 
changed slightly in that Julie ended up doing much of the analysis manually rather than 
with a computer program she had intended to use. She was much clearer at the end of the 
process than at the proposal stage regarding how she approached data analysis. Initially I 
had some difficulty making sense of her handwritten notes on the evolution of the 
dissertation chapter outlines and her extensive, typed coding system. However, when 
Julie and I met she walked me through the process and materials pertaining to analysis 
and specifically outlined the steps she used. I recorded notes as Julie talked and 
suggested that she transfer the specific analytic steps she had just articulated to Chapter 
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Three of her dissertation, so that the reader was very clear regarding the analytic steps 
used. She incorporated this suggestion into the next draft of her dissertation. 

2) During the audit meeting with Julie I noted the accuracy of transcription of interviews. 
I listened to seven tapes including at least one interview with each primary informant, one 
interview with a secondary informant, and one dialogical reflection. With pencil in hand, I 
noted inaccuracies in the transcripts. I listened to at least five pages of each interview (a 
total of 38 pages) and noted as few as two errors and as many as fifteen errors per 5-7 
page set. Overall, transcription was exceptionally accurate. The inaccuracies identfied 
were very minor, including single words that were omitted or mistranscribed and/or short 
phrases that were omitted. In my estimation, the minor inaccuracies did not impact the 
overall content of the transcriptions. An example of the inaccuracies identified in one set 
of 5 pages is as follows (excerpt from an interview with Tom on 5/26/94): 

Page# Error in transcription 

1 no errors 

2 the word "dies" should be "died" 

2 phrase "in a bucket" was omitted 

2 the words "particular use" should be "curriculum" 

3 the word "the" was omitted 

3 the words "other things" should be "only thing" 

4 phrase "the beginning of' was omitted 

5 the word "there's" was omitted 

6 the words "image of' were omitted 

6 the word "new" was omitted 

6 the word "even" was omitted 

7 the word "but" should be "buy" 

3) It appears that the trustworthiness of the study can be established in that the findings 
seem to be clearly grounded in the data. The researcher carefully designed her project and 
employed a number of verification strategies (including member checks, triangulation, 
prolonged observation, dialogical reflection with informants, and external audit) to ensure 
the accuracy of the data. The data were presented in detail in Chapters IV through VI 
using extensive informants' quotes and descriptive language. Having reviewed excerpts of 
the transcripts, observation protocols and field notes and compared them to the 
dissertation draft, it appears that the data accurately represents the informants' 
perspectives. The conclusions, discussed in Chapter VII, flow logically from the data 
presented in Chapters IV through VI. Comparing those chapters to the transcripts, 
documents, and analytic methods, the conclusions of the study seem warranted. 
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External Audit 
Attestation 

by Dana L. Miller, Ph.D. 

Robert Franzblau requested that I complete an educational audit of his qualitative 
dissertation titled: Aesthetic Education as a Subversive Activity: A Phenomenological Case 
Study of Robert Kapilow . The audit was conducted between March 22-24, 1996. The purpose 
of the audit was to ascertain the extent to which the results of the study are trustworthy. 

In their book Naturalistic Inquiry, Lincoln and Guba (1985) suggest that the audit "may 
be the single most important trustworthiness technique available to the naturalist" (p. 283). The 
educational audit is "based metaphorically on the fiscal audit" (p. 317). The role of the auditor is 
to carefully examine both the process and product of the inquiry. In order to accomplish an 
audit, it is imperative that the researcher maintain careful, detailed records throughout the inquiry. 

Lincoln and Guba (1985) delineate two tasks in the audit process: 

1) examination of the PROCESS of the inquiry to ensure that informants are represented fairly in 
recorded accounts, and 2) examination of the final PRODUCT to ensure accuracy; in particular 
that the findings are supported by the data. 

To meet the outlined purpose of this audit, numerous materials were reviewed. The 
researcher submitted a box with 30 file folders containing several hundreds of pages of data and 
documentation of the research process, neatly organized and labeled. He also submitted an audio 
cassette case containing 33 audio tapes, each labeled and dated. I examined and documented the 
contents of each folder. The following materials were submitted for the audit: 

1) A NODES list (2 page list and diagram, based on the use of NUD.ist 

data analysis software) 

2) NODE reports (text retrieval function, NUD.ist software) for interviews 

(folder contained 1 8 reports of varying lengths, approximately 400 pages) 

3) NODE reports for observations (approximately 75 pages) 

4) NODE reports for thematizing/structural descriptors/metaphors 

(approximately 200 pages) 

5) Interview transcripts (three transcripts, transcribed verbatim, single spaced on 8 1/2" x 11" 

paper. Interviews were conducted in the executive suite of a hotel in New Jersey in late 
July, early August, 1995. Total pages of transcript=75) 

6) Interview protocol development. This folder contained approximately 22 pages of potential 

interview questions. It also contained three interview protocols labeled: 

Part I: Focused Life History (1 pg, 15 questions) 

Part II: Details of Experience (3 pg, 29 questions) 

Part III: Reflection on Meaning (2 pg, 18 questions) 
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4) Although the issue of "reliability" is problematic for qualitative researchers and 
generalizability from specific cases may be limited, the results of this audit demonstrate 
that the researcher maintained an excellent audit trail. It was apparent in reviewing the 
materials that Julie had been rigorous and systematic in her data collection and record 
keeping. The materials presented for the purpose of the audit were detailed, complete, 
and well organized. Other researchers could follow this researcher's clear audit trail. In 
many ways Julie's study reflected the intimate involvement of informants in the study, 
and Julie's intimate involvement at the site and with the primary informants. In addition, 
the findings of the study are presented with rich, thick description to enable readers to 
make the determination regarding whether the results might be transferable or comparable 
to their settings. 

Attested to by Dana Miller this 27th day of March, 1995. 

Dana L. Miller, Ph.D. 

Assistant Professor (Adjunct): Qualitative Approaches to Research in Education 
Department of Educational Psychology 
University of Nebraska-Lincoln. 
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External Audit Attestation - Page 2 

7) Researcher Journal (27 journal entries dated from 1 1/15/94 through 1/30/96. 

Entries were brief, a total of 22 pages, single spaced on 8 1/2" x 1 1 " paper. In his journal, 
the researcher chronicled his research activities and discussed methodological issues. In 
later entries he discussed coding procedures, techniques he tried, things that did not work. 
He documented his constant questioning, "What is going on here?" In reference to data 
analysis, he discussed his attempt to find appropriate labels that fit and came close to 
Kapilow's meaning. The researcher also initially outlined the chapters of his dissertation 
and content to be included in chapters four, five, six and seven. I verified this against the 
dissertation and the researcher's journal entry accurately reflected the 
structure of the final product. 

8) Kapilow Correspondence (a series of six letters the researcher wrote to Kapilow, beginning 

with the initial letter of inquiry regarding Kapilow's interest/willingness to participate in a 
research study, and including follow-up correspondence after the researcher's trip to New 
Jersey to conduct interviews was concluded) 

9) Fieldnotes-Original (Handwritten fieldnotes from observations, on 8 1/2" x 1 1" paper, plus 

schedules of sessions/workshops observed). Twenty eight observations were recorded— 
one to two pages each- fora total of 33 pages of fieldnotes. Sessions were labeled 
MusiConnection I, II and III; with the date and name of the school/radio station/rehearsal/ 
performance on each entry. Entries were dated from 9/18/95 through 2/12/96. After I 
cataloged the materials in this folder, I compared them to the researcher's documentation 
in his dissertation, and found his records to be accurate (Kapilow.. ."was observed at 28 
educational sessions over the course of six months" p. 134 of Franzblau's dissertation) 

10) Fieldnotes - Transcribed 

1 1) Audio Cassette Tape List (6 pages total, all tapes were listed by date and name of 

event/performance/interview). I checked the researcher's documentation against the titles 
and dates on the 33 tapes submitted for the audit, and found the researcher's records to be 
accurate. 

12) Articles-Kapilow (approximately 75 pages of photographs, newspaper articles, reviews, 

press releases) 

13) Informed Consent (documentation of the researcher's request for IRB approval-under 

exempt status, the IRB's approval, signed consent by Kapilow) 

14) Green Eggs and Ham Project (correspondence, posters, press releases, and media articles 

related to this project; music written by Kapilow, performances in Lincoln, NE conducted 
by the researcher) 
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15) Data Analysis Notes (approximately 30 pages, some handwritten, diagrams, typed lists of 

codes, chapter outlines, codes with verbatim examples) 

16) Tapes Correspondence (approximately 5 letters to individuals in New York and Washington 

D C., requesting tape recordings of Kapilow's performances-with his permission. Their 
responses and materials sent to the researcher were included) 

17) KC (Kansas City) Correspondence (related to observations of MusiConnection) 

18) KC (Kansas City) Friends of Chamber Music (Program— 20th anniversary season, 1995- 

1996; 148 pages including sections on MusiConnection and Kapilow's "What Makes it 
Great" series) 

19) Copyright-Seuss Estate 

20) Kapilow in Lincoln (34 pages of transcript from KUCV Radio, Interview with 

Kapilow on 1 0/26/94) 

21) Fieldston School 

22) New Jersey Opera, Chamber Music 

23) Kapilow's program notes (notes used to introduce Beethoven concert) 

24) Contacts (names, addresses and phone numbers, ahanddrawn map) 

25) Outline: Pilot #2 (Interview questions, protocol and informed consent for a pilot interview 

with T. White) 

26) Notes from Dexter (1970) 

27) Kapilow Press Kit 

28) Preliminary Drafts-Proposal (Proposal was 32 pages in length. Five drafts were included in 

this folder; one submitted to the researcher's seminar class, and four to faculty on his 
committee. Drafts had handwritten notes/suggestions for revisions throughout) 

29) Musical examples, figures 

30) Preliminary Drafts-Dissertation Chapters (10 drafts of various chapters of the researcher's 

dissertation, with scribbled notes from faculty throughout) 
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11 ). 

2) Data collection procedures followed those proposed in the methods section of the proposal. 
The researcher proposed a series of three, ninety minute interviews, focusing on the participant's 
a) background, b) experiences, and c) meaning. The transcripts submitted for review documented 
the interview process. The researcher was consistent in his use of three interview protocols to 
guide the interview. I determined, however, in comparing the proposed protocols to those used, 
that the protocols used during the interviews were much more extensive than those proposed. 
This provided evidence that the researcher continued to develop and refine his thinking 
throughout the project. In reviewing the transcripts, I also concluded that the researcher did an 
excellent job of probing for additional information when appropriate. In addition, the researcher 
proposed to conduct a series of observations at educational sessions the participant was 
conducting. The researcher attended and clearly documented 28 observations. He also used 
several written materials (documents) and tape recordings of lectures and performances as 
supplemental data. Data collection procedures are clearly documented in the dissertation (pgs 
20 - 21 ). 

3) Data analysis procedures followed those proposed in the methods section of the 
proposal. In particular, the researcher discussed the importance of avoiding a priori theorizing 
and bracketing preconceived notions, and discussed the use of NUD.ist, a computer software 
program to aid in data analysis. Data analysis procedures are clearly documented in the 
dissertation (pgs 22-23 and in Chapter Five). 

4) The initial structure proposed for the dissertation (outlined on pages 27-28 of the proposal) 
changed. The researcher proposed writing nine chapters, and instead organized the dissertation in 
seven chapters. This change, too, is reflective of the qualitative nature of the inquiry and was 
appropriate for this project. 

5) It appears that the trustworthiness of the study can be established in that the findings seem to 
be clearly grounded in the data. The researcher carefully designed his project and employed a 
number of verification strategies (including member checks, triangulation, prolonged observation, 
and external audit) to ensure the accuracy of the data. The data were presented in detail in 
Chapters Four through Seven using extensive quotes and descriptive language. Having reviewed 
excerpts of the transcripts, interview and observation protocols and field notes and compared 
them to the dissertation, it appears that the data accurately represents the participant’s 
perspectives and experiences. 

6) Although the issue of "reliability" is problematic for qualitative researchers and 
generalizability from specific cases may be limited, the results of this audit demonstrate that the 
researcher maintained an excellent audit trail. It was apparent in reviewing the materials that 
Franzblau was rigorous and systematic in his data collection and record keeping. The extensive 
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The audit consisted of the following steps : 

1) I read the dissertation proposal, dated Spring, 1995. 

Particular attention was paid to sections addressing the purpose of the study, 
research questions, phenomenological/case study design, proposed data collection and 
analysis methods, verification strategies, the researcher's role in the study, and the 
interview protocols. 

2) I cataloged and reviewed all materials submitted for the audit. 

3) I read all methodological/procedure sections of the dissertation and much of the final chapters 

presenting the data, discussing themes, providing synthesis, conclusions, and 
recommendations. I compared the original dissertation proposal (focusing on the purpose 
of the study, research questions, proposed methods) to the dissertation. 

4) I reviewed research methods, analysis, and verification procedures and compared the 

researcher's stated procedures with the audit trail he established through the 
documentation submitted for the audit. 

5) I read several pages of transcribed interview data. I then compared the interview protocols 

against the verbatim transcripts to determine whether the researcher had followed his 
proposed protocols during the interviews. 

6) I wrote and submitted the attestation. 

Following completion of the preceding steps, this auditor submits the following 
conclusions : 

1) It is this auditor's opinion that the focus of the study remained consistent with 
the proposed focus. The researcher did, however, slightly reframe his dissertation title and 
research questions as he further refined the focus of his study. For example, the proposal was 
titled, A Phenomenological Case Study of Robert Kapilow: Exploring the Intersection of Music 
Education and Conducting . The dissertation title reads: Aesthetic Education as a Subversive 
Activity: A Phenomenological Case Study of Robert Kapilow . This change was appropriate, and 
was clearly data driven, given the meaning the study participant placed on subversive education 
and its influence in his life and teaching. The researcher also changed from two grand tour 
questions and six sub-questions in the proposal, to six research questions in the dissertation. 

The questions posed in chapter one of the dissertation were similar, but seemed to have a broader 
focus. These revisions were not a concern, in fact are to be expected given the nature of 
qualitative research as an emerging design. The researcher articulates this in his dissertation: 
"Research questions remain flexible and open to modification throughout the course of study" (p. 
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materials presented for the purpose of the audit were detailed, complete, and well organized. 
Other researchers could follow this researcher's clear audit trail. In many ways Franzblau's study 
reflected the intimate connection he had to the participant and his music. In addition, the findings 
of the study are presented with "rich, thick description" to enable readers to determine the 
applicability of this study to their setting(s). 

Attested to by Dana Miller this 24th day of March, 1996. 



Dana L. Miller, Ph.D. 

Assistant Professor 

900k Qualitative Approaches to Research in Education 
Department of Educational Psychology 
University of Nebraska -Lincoln 
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Types of Materials Submitted for an External Audit: 

Note: The following is a synthesis of types of materials submitted for an external audit by eleven 
graduate students from April 1994 through November 1996. Materials have been assigned to 
“audit trail” categories #1 through #6, identified by Lincoln and Guba (1985) in Naturalistic 
Inquiry, and #7 and #8, my addition, which emerged from an analysis of material submitted. 

1) Category: Raw Data 

• interview transcripts (labeled, dated) 

• interview audio tapes (labeled, dated) 

• observation notes (field notes, observation protocols) (labeled, dated) 

• participant j oumals 

• participant art work 

• demographic survey data collected on participants 

• photographs taken during field work 

2) Category: Data Reduction and Analysis Products 

• coded transcripts 

• index cards identifying codes, themes, and related data chunks 

• narrative description of the analytic structure of a product, and emerging process 

• lists of codes/themes/categories 

• envelopes and/or file folders containing excerpts of transcripts/data relevant to codes/ 

themes 

• Computer printouts/Node reports displaying data analysis structures 



3) Category: Data Reconstruction and Synthesis Products 

• sections, chapters, entire dissertation products 

• drafts of emerging theoretical propositions, models and refinements 

• visual diagrams/data displays 

4) Category: Process Notes 

• research log, chronicling all research activities 

• telephone logs related to research activities 

• researcher journals 

• correspondence between researcher and participants 

• correspondence between researcher and gatekeepers 

• feedback on emerging product from faculty advisor/readers (and in some cases notes 

regarding how the feedback was integrated) 

• description of the evolution of the dissertation chapter outlines 

• documentation of methodological sources used 

• research course notes applicable to the research process 

• research timelines and chronologies (for data collection, analysis, writing) 
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Types of Materials Submitted for an Audit (Continued) 

5) Category: Materials Related to Researcher Intentions/Dispositions 

• the dissertation/study proposal 

• notes regarding suggested methodological revisions from faculty/committee meetings 

• Institutional Review Board materials (request for permission, outline of procedures, 

documentation of approval, consent forms) 

• documentation of site access (requests, explanation of purpose/methods, permission) 

• documentation of framework for developing the written structure/format for composing 

the final report 

• documentation of sampling criteria and process 

6) Category: Information Relative to Instrument Development 

• drafts of interview questions/instruments 

• lists of possible areas of inquiry for interviews 

• copies of interview protocols 

• protocol revisions/refinements as data collection evolves 

7) Category: Verification/Validity Documentation 

• attestation or notes from peer reviewer(s) 

• member check forms, notes, transcripts, correspondence returned to researcher 

• documentation of member check conversations with participants 



8) Category: Materials Prepared for the Auditor/Audit 

• catalogue of materials submitted for the audit 

• memos to the auditor regarding researcher decisions, intuitive processes, how the study 

framework emerged, and/or about the materials/how they were organized 

• memos documenting “random thoughts” and “miscellaneous issues” 

• notes to the auditor regarding revisions related to areas of concern identified in the 

initial audit process 
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